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ABSTRACT 

The two studies reported examined the validity of 
several Evaluation/Survey Service (ESS) surveys for accurately 
reflecting changes in students' perceptions resulting from changes 
made by an institution in its policies, programs, services, or 
environment. In study 1, personnel from nine pos tsecondary 
institutions identified items on various ESS surveys for which 
student responses were expected to change as a result of changes made 
by the institution since the last survey administratioii. Actual 
changes were compared to projected changes, and mean student 
responses were compared for different administrations of a survey. In 
study 2, changes in mean student responses were identified for two or 
more administrations, and personnel were asked to provide 
explanations for these changes. Overall, the two studies strongly 
support the validity of the following three ESS instruments for 
reflecting changes in students' perceptions over time: (l) the 
Student Opinion Survey (4-year); (2) the Survey of Academic Advising; 
and (3) the Wi thdrawing/Nonreturning Student Survey. Three other ESS 
surveys received less support, perhaps because of the smaller sample 
sizes associated with these surveys. Four tables present response 
data. Six appendixes present additional information about study 
methodology, including cover letters and survey forms, and responses. 
(Contains 12 references.) (Author/SLD) 
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ABSTRACT 

The two studies reported here examined the validity of several 
Evaluation/Survey Service {ESS) surveys Cor accurately reflecting changes in 
students' perceptions resulting from changes made by an institution in its 
policies, programs, services, or environment. 

In Study 1 we asked personnel at several post secondary inst itut ions to 
identify specific items on various ESS surveys for which student responses were 
expected to change as a result of changes made by the institution since the last 
survey administration. Changes in actual student responses were compared to 
these projected changes; the ratio of hits to misses was analyzed and mean 
student responses were compared for different administrations of a survey. 

In Study 2 we identified changes in mean student responses over two or more 
administrations and then asked institutional personnel to provide possible 
explanations for those changes. Each explanation was rated in terms of the 
likelihood that the suggested institutional change could account for the observed 
differences in student responses. For the four surveys examined, 74 percent of 
the proposed explanations v/ere considered acceptable. 

Overall, the two studies strongly supported the validity of three ESS 
instruments for reflecting changes in students' perceptions over time: the 
Student Opinion Survey {4-year) , the Survey of Academic Advising, and the 
Withdraw/Nonreturning Student Survey. Three other ESS surveys, the Adult Learner 
Needs Assessment Survey, the Student Opinion Survey (2-year) , and the Entering 
Student Survey, received less support, perhaps due to the smaller institutional 
and student sample sizes associated with these surveys. 
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THE VALIDITY OF EVALUATION/ SURVEY SERVICE SURVEY 
INSTRUMENTS FOR REFLECTING INSTITUTIONAL CHANGE 

The Evaluation /Survey Service (ESS) surveys were developed in the late 
1970 's to provide educational institutions with the means to assess students' 
opinions, attitudes, goals, and impressions. These instruments offer several 
=^dvantages to institutions, including theory-based construction, availability of 
consultation with expert practitioners, pilot tested items, ease of 
administration and processing, and the availability of a variety of user-norm 
groups . 

The effectiveness of these surveys depends upon the degree to which they 
meet appropriate standards of validity and reliability (i.e., are they 
appropriate for their intended uses and do they provide consistent and stable 
measurement) . Although each survey serves a slightly different purpose, they all 
provide student information that administrators can use to help guide and 
evaluate institutional reform. 

Background 

Although the reliability of many ESS surveys has been examined (e.g., 
Valiga, 1983 ; see also the ESS User's Guide, 1989) the validity of these 
instruments for specific uses has been investigated primarily only through local 
validity studies. These studies have been conducted at individual institutions 
to determine the degree to which information from a particular ESS survey could 
help them improve their services or programs. ESS surveys examined in these 
local validity studies include the Withdrawing/Non-returning Survey (Granger, 
1981; Nelson Gt Urff, 1982), the Alumni Survey (Jones, 1982), and the Student 
Opinion Survey (Cosgrove, 1984; Klainer, 1982). Although these studies 
consistently found that particular ESS surveys provided useful feedback from 
survey respondents, generalization of the results to other institutions with 
different environmental and student characteristics could not be assured. 

A few multi-institution studies have addressed the validity of ESS survey 
instruments for identifying institutional characteristics that contribute to 
student success. Forrest (1985) examined responses to the Alumni Survey from 
recent graduates of 40 institutions and found positive relationships among 



7 



2 

graduate satisfaction, rates of persistence to graduation, and an individualized 
instructional style. 

Valiga (1980) conducted a factor analysis on responses to the Student 
Opinion Survey and found a positive relationship wiuii a structure of college 
outcomes developed at the National Center for Higher Education Management 
Systems- Valiga (1982) also conducted a factor analysis of responses to the 
Student Opinion Survey from students at 42 institutions and found a factor 
structure that was highly similar to the six subgroups of satisfaction-ratings 
items in that survey. 

Davis (1982) investigated the discriminant validity of the Adult Learner 
Needs Assessment Survey and found that this instrument was capable of 
distinguishing among the personal and career needs of older adults, young adults, 
and traditional-aged students. 

The preceding studies provide some support for the validity of the ESS 
surveys examined, primarily as instruments for eliciting the perceptions of 
students concerning institutional programs, services, or general environment. 
However, no studies examined spt j.fically the validity of ESS instruments for 
assessing changes in student perceptions over time. The purpose of the present 
study was to examine the degree to which ESS surveys accurately reflect changes 
in student perceptions resulting from changes made by an institution in its 
programs, services, and/or environment. An instrument that is valid for 
measuring students' perceptions of an institution (e.g., a survey) will obtain 
accurate and consistent results over time, as long as those perceptions remain 
stable. If an institution implements a change or reform to a particular program 
or service, students' perceptions should change accordingly. If the survey is 
valid for measuring changes in student perception , these changes wi 11 be 
reflected by changes in students' responses to relevant survey items. 

This study investigated the validity of selected ESS surveys for reflecting 
changes in students'' perceptions over time. These perceptual changes were 
assumed to have resulted from modifications or reforms in institutions' programs, 
services, or environment. Two questions were examined: 
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1. Do the survey items reflect changes in student perceptions projected 
by institutional personnel , as measured by differences in mean 
student response over time? 

2. Can changes in student perceptions over time, as measured by 
differences in mean student response, be explained after the fact in 
terms of specific institutional changes/reforms? 

These questions were addressed in two separate studies. Both studies examined 
the capability of relevant survey items to reflect perceptual changes over time. 
Study 1, however, required institutional personnel zo predicc perceptual changes, 
based on institutional reforms that had been implemented. Study 2 asked 
institutional personnel to explain existing differences in mean student response 
over time in terms of institutional reforms, if possible. 

Analysis of Projected Difforoncea 

Data for Study 1 

We identified sixteen ESS user-institutions that requested a particular 
survey, and that had administered the same survey between one and three years 
earlier. Personnel from these institutions were asked to complete a 
questionnaire sent immediately following their current order. They v;ere asked 
to report any institutional changes made since the last administration of the 
survey that might influence students' survey responses on the next 
administration. We then asked them to identify specific items on the survey that 
they felt would be affected by these reforms and to predict the nature of the 
changes in student response. For example, one might predict that opening a new 
computer center would increase students' ratings of satisfaction with computer 
services- Personnel were asked to return the completed questionnaires prior to 
obtaining the results from their next survey adn^inistration. A copy of the cover 
letter and questionnaire used in the study are provided in Appendix A. 

The usability of the responses from institutional personnel was evaluated 
using several criteria: First, each proj ected difference in student respt-nse had 
to involve data not yet collected. Predictions that failed to meet this 
criterion were removed from this study and added to the analysis of explained 
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the projected response changes had to be 
ers predicted opposing changes in mean student 
fed on two different institutional reforms. 
i they were unsure which direction a change in 
iled to specify the direction. Where such 
lal personnel were contacted by phone and by 
:he ambiguity. If these ambiguities remained 
• that institution v;ere dropped from the study. 
30 students per institution was required foi 

Institutions returned usable predictions and 
utions provided data for the Student Opinion 

two provided data for the Survey of Academic 
or the Adult Learner Needs Assessment Survey. 
S 4 -Year) . The SOS 4-year examines enrolled 
■grams, services, and environment provided by 

survey form comprises five sections: Section 
,a . Section II gathers student ratings or usage 
Lng 23 types of college services and programs 
:ural programs} . Section III collects student 
'liege environment (academic, admissions, rules 
fistration, and general). SectionSF IV anu V 
il items and for comments from respondents, 
contains the demographic data for the six 
or the SOS 4-year. Five of these institutions 
s, and one offered a professional degree (e.g. , 
rvey administration procedures and sampling 
:>ng institutions, but were, for all but one 
ninistrations within each institution. 
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Table B2 of Appendix B summarizes the student sampling procedures used by 
each institution that administered the SOS 4-year. The time period between 
survey administrations ranged from one to three years, but usually was between 
one and two years. The surveys were administered to all four undergraduate 
classes for all but one of the six institutions. 

Student characteristics that might influence responses to some survey items 
also were examined for each institution (see Table B3 of Appendix B) . These 
characteristics included age, race, sex, marital status, purpose in attending the 
institution, and college residence. Again, differences over time within a 
particular institution were generally minor. The largest intra-mstitutional 
differences were found for the percent of students living on campus; although all 
six institutions showed a decrease in the percent of students living on campus 
between the tv;o administrations, the size of the decrease showed some variation, 
ranging from 2% to 19% across institutions. 

Survey of Acadoinic Adviging (SAft) . The SAA obtains students' impressions 
of their institution's academic advising services (as distinguished from personal 
or career counseling services) . The survey form is composed of 7 sections, 3 ot 
which provided data for this study: Section I collects student biographical 
information. Section III assesses the degree to which students have discussed 
18 types of topics with their academic advisors along v;ith their ratings of 
:5atisf action with their advisor's assistance for each topic discussed. Section 
IV asks students to rate their level of agreement with 36 statements about their 
advisor (eg., My advisor knows who I am; My advisor allows sufficient time to 
discuss issues or problems.) 

The two institutions that returned data for the SAA were both eastern 
colleges with less than 3,000 students. One was a two-year suburban community 
college offering an Associate degree program, and the other was four-year college 
offering a Bachelor's degree program. 

Student characteristics that might influence responsetJ to some survey items 
v;ere examined for each institution. These included age, race, sex, college GPA, 
and purpose for attending the institution. As shown in Appendix C, both student 




toL" each mst x iiut* loii . 

Adult Loarnor Nacdg AggQSginont Survoy (ALNAS) . The ALNAG r^yploi.-.: ^hn* 
porceived .><Juccit lonal and p^i'sonal need? of ^?nroll^^d ..md pi osp^^cT i\'h uiuit 
studenCG. The i^urvey form comprises S seccions, 2 of which pLovid«rd M ci t'>i 
this study: f^^ection I collectE demographic data from :;tud*?nt3 and Z'^rv if\i\ Ii: 
A^kc^ students to fate th*? d*^gi'e«^ ro which tliey n»eed heli,. v.'ath -•ach .^t r ■ - , p..'L jni..i i. 
•)i '^d'jcat lona i needs lu the areciL^ of iife uiu ■a-^-i' i-v-- 1 .frn-i.^ 

educational planning, and associatioi'is with others*.. 

The one inGti*:ution that returned survey data tor fb-r? ALMA.? v/as a tour v-ar 
:;tate college in f he eastern United States, offering Vxjcn E.ach»^lor- ^ ■^r,d Ma-.u -i ■ 
degree programs to approximaf ely 3,600 student-. Kei-^v.^nr .rua-^i' 

characteristics as v;ell as a description of the samplmq proc^dure-9 vir.eci t<^i -M.-h 
of three administrations are provided m Appendix D. Thren administrat loin: v.'.^r-^ 
included for this survey because institutional personnel indicated that '^awt 
changes had been started betv;een the first and second adminici: rar lonts , i-ut t-|i-'ii 
effects were expected to develop gradually. Although the total c^ampl^^ . ine v. i.- 
about J 00 students for each administration, this nurnb»^r 'lecrn^asf^d to rhan 
30 students for some items. This reduction occurred because i^tudent.-j maiK-^d .i 
"Does Mot Apply" option in appropriate .-situations (e.g., "I need help copinM v."ith 
divorce or separation" > . 
Mothod 

Data that met the usability criteria outlined earlier in this paper v.vi»^ 
analyzed by first computing the mean student response for each item by 
administration, and then calculating the difference between pre- and post-chanoe 
mean responses (i.e., across administrations). These differences v;ere compared 
with their respective predictions. Differences betv;een the tv;o responses weie 
designated as hits if they were in the expected direction, or as misses if they 
were in the opposite direction. We then used the Sign Test for Matched Pairs 
(Hays, 1981, p. 537) to determine v;hether the proportion of hits to mis^e;- wa^ 
significantly greater than the proportion that would be expected due to chance 
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alone {£ ^ .001) . Finally, we conducted a two-sample t test for each target item 
to determine whether the difference in mean student response was statistically 
significant . 

ROBUltS 

Student Opinion Survov - A -Year > The six SOS 4-y9ar institutions 
identified a total of 31 items for which they anticipated changes in mean student 
response. Five of these items were eliminated; four items were discarded because 
the institutional changes occurred after the most recent administration, and one 
item was dropped due to small sample size (N < 30) . Three of the five discarded 
items came from one institution, representing 75% of the targeted survey items 
for that institution. The other two discarded items came from two different 
institutions, and represented 5% of the total number of targeted items from one 
institution, and 17% of the total from the other institution. The final item 
pool consisted of 26 items for which changes were anticipated between pre-change 
and post-change means. 

Table 1 indicates that 23 of the 26 projected changes in mean student 
response were supported (hits). The Sign test indicated that this level of 
agreement differed significantly from chance (z = 3.23; ^ < .001) . The C tests 
for each item revealed that mean student responses differed significantly over 
time for 12 of the 23 hits (£ < .05). 

survey of Acadaxnic Advising > Table 2 contains the results of the analysis 
for the items from the Survey of Academic Advising. Personnel at the two 
institutions identified 23 items for which they anticipated changes in mean 
student response. Eighteen of the 23 items showed changes in mean student 
response that were in the expected direction (hits) . For three of the remaining 
five items, changes were in the opposite direction from what was expected 
(misses), and the other two items showed identical mean student responses for 
both administrations (ties). The Sign test for the 21 untied pairs of means 
indicated that the level of agreement (hits) differed significantly from that 
expected due to chance alone (z = 5.68; £ < .001) . We conducted a series of t 
tests and found significant differences between mean student responses for 5 of 



the 18 hits ^£ - .05). Differences for three additional hits were significant 
at a less restrictive level of significance (j^. ^ .10). 

Adult Learner Needs Aasasament Survey* Table 3 contains the results of the 
analysis of ALNAS items. Institutional personnel identified 22 items for v/hich 
changes m mean student response were expected. Two of th^se items were 
discarded due to small sample sizes (N < 30) . We compared differences in mean 
student response for the remaining 20 items and found nine hits, nine misses, and 
two iiies. The Sign test for the 18 untied pairs showed that the ratio of hits 
to misses did not differ significantly from chance (£ > .05) . We conducted t 
tests for each of the 18 untied pairs, and found significant differences in mean 
student response for tv;o of the nine hits < -OS). 

Analysis of Explained Differences 

Data for Study 2 

The data for this i?tudy were obtained in two ways. We first identified 187 
institutions that had administered the same survey more than once to their 
students. For each institution we compared the type of student samples, 
administration techniques, and sampling techniques used for each administration 
uo determine their similarity over time. Institutions were eliminated if they 
used nonrandom sampling or if they administered the survey forms less than one 
year apart. In some cases an institution administered the survey forms at 
intervals of less than one year, but had continued this process over several 
years. When this occurred, item response comparisons were made only at one-y^ar 
intervals (e.g., March, 1986 responses would be compared to March, 1987 
responses) . This procedure resulted in a sample of 59 institutions. The second 
source of data consisted of 28 items from five institutions participating in 
Study 1. These items showed relatively large differences over time, but had not 
been identified ^ f lagged) by institutional personnel as items for which they 
anticipated changes. The final combined sample consisted of responses from 64 
institutions that had administered a total of 68 survey instruments at least 
twice ( including four institutions that administered two different surveys 
twice . ) 
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For each institution and survey administration, mean student responses were 
comput&d for all LiJcert-type items. Differences in mean student response were 
then computed for each item, across survey administrations. Differences were 
identified in accordance with the following criteria: Mean differences of .35 
or greater were required for 5-option items, differences of .30 or greater were 
required for 4-option items, and differences of .25 or greater were required for 
3 -option items. (These somewhat conservative criteria were used to ensure that 
mean differences would be both statistically ana meaningfully significant given 
a considerable range of sample sizes from institution to institution.) In cases 
involving sample sizes less than 100, required minimum mean response differences 
vvere increased by .05 units for all items. For each institution, items that 
showed the greatest differences between administrations were selected; a maximum 
of five items were used for each institution. 

Questionnaires were sent to the 59 institutions in May, 1987. We asked 
institutional personnel to identify changes or reforms that might have resulted 
in the observed differences in mean student response. They were asked to 
describe those changes, and to provide the dates they occurred. A copy of the 
cover letter and of the questionnaire are provided in Appendix E. A follow-up 
letter was sent one month later to non-respondents. Three weeks later copies of 
the original questionnaire and a revised cover letter were mailed to each 
institution that still had not responded. Letters were also sent to the five 
institutions from Study i concerning the 28 unf lagged items that had shown large 
differences in mean student response. Institutional personnel were asked to 
identify any change or reform made at the institution that might account for the 
differences in mean student response, and to include the date of each change. 

Questionnaires were returned by 26 of the 59 institutions selected 
specifically for Study 2, and by all 5 of the institutions from Study 1, 
resulting in an overall response rate of 51%. Responses were received for four 
surveys : 

(i) the Student Opinion Survey for 4 -year colleges (SOS 4-year), which 
explores students' perceptions of the programs and services offered at 
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their institution. 

(2) the Withdrawing/Nonreturning Student Survey (W/NRSS) , which helps 
institutional personnel determine why some students leave before finishing 
a degree or certificate program. 

(3) the Student Opinion Survey for 2~year colleges (SOS 2~year), which is 
similar to the SOS 4-year, but tailored to meet the special needs of tv;o- 
year institutions . 

(4) the Entering Student Survey (ENSS) , which provides a variety of 
demographic, background, and educational information about students who 
are newly enrolled at an institution. 

Appendix F contains institutional and student characteristics and a summary 
of the sampling procedures used by the 31 institutions that provided data for 
Study 2. The 21 institutions that used th" Student Opinion Survey ( 4-year > 
represented a diverse sample of geographical regions across the United States. 
Additionally, these institutions represented a broad range of both institution 
and community sizes, types of degrees offered, affiliations, and academic 
programs offered. Participating institutions for the other three surveys were 
considerably fewer in number, and thus reflected a somewhat smaller range of 
characcerist ics . However, characteristics of the student population seldom 
differed by more than 10% from one administration to the next within any 
particular institution, and were considered unlikely to influence the results. 

In most cases, sampling procedures were similar across administrations Lor 
a particular institution. Student response rates were relatively low or 
inconsistent for some institutions, particularly for the W/NRSS and SOS 4-year 
surveys. As a result, mean student response may be less representative of the 
total student population at these institutions. 
Mathod 

The explanations of mean response differences provided by institutional 
personnel were first examined for clarity. Ambiguities were resolved through 
further discussion by phone with institutional personnel. Next, each explanation 
was categorized in the following manner: 
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Not acceptable: No explanation given. 

b. Not acceptable: The explanation implied a change in the opposite 
direction from the data. 

c. Not acceptable: The explanation addressed the wrong time period or 
the v;rong content . 

d. Not acceptable: The explanation was judged too subjective. 

e. Acceptable. 

Two raters separately categorized each explanation using this rating scheme, and 
the two sets of ratings were compared. The raters categorized 131 of the 136 
explanations identically. For three of the remaining five explanations, both 
raters described the explanations as not acceptable, but differed in their 
reasons for this rating. Thus, the raters differed in their ratings of only 2 
of the 136 explanations in terms of their acceptability. All fiv^' discrepancies 
in ratings were resolved through discussion. 

Hit rates were calculated for each item by finding the ratio of the total 
number of institutions providing acceptable explanations for that item to the 
total number of institutions providing any explanation for that item. Table 4 
contains a list of the number of acceptable and unacceptable explanations and the 
hit rates for the relevant items in each of the four surveys. 
Results 

Student Opinion Survey (4-yoar) . Overall.- 75 of the 102 explanations 
produced by the institutions were rated acceptable, yielding a hit rate of 74%. 
Fourteen items from Section II of the survey {representing 61% of the items m 
that section) v;ere analyzed. Of the 44 explanations provided for these items, 
34 were rated acceptable, resulting in a hit rate of 77%. For Section III, 28 
items were analyzed, representing 67% of the items in that section. Of the 58 
explanations provided for these it<^ims, 42 were rated acceptable, producing a hit 
rate of 72%. 

Of the 42 items for which explanations were analyzed, 26 items had 100% hit 
rates and 6 items had 0% hit rates. Satisfactory explanations v;ere generally 
available for all major aspects of the college environment covered in the SOS (4- 
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yean, with the exception of "campus rules and regulations." Items involving 
this topic (e.g., III-17 and III-18) showed hit rates below 50%. 

Withdrawing /Nonretuming Student Survey ♦ Eighteen explanations were 
provided for a total of 16 items on the W/NRSS. Four of the explanations 
concerned four items in Section II (8% of the items in that section); three of 
the four explanations (75%) were rated acceptable. The remaining 14 explanations 
concerned 12 items from Section III (about 26% of the items in that section) . 
Eleven of these 14 explanations (79%) were rated acceptable. 

Student Opinion survey (2 -year) ♦ The explanations for the SOS ( 2 -year ) 
concerned 10 items from Sections III and IV, representing 16% of the total number 
of relevant survey items. Seven of the 10 explanations provided (70%) were rated 
acceptable . 

Entering Student survey ♦ Explanations for the ENSS were provided for six 
items, representing 13% of the items in Section III of the instrument. Three of 
the six explanations provided (50%) were rated acceptable. 

DiscusBion 

Summary 

Two approaches were used to examine the validity of ESS surveys for 
reflecting changes in students' perceptions resulting from institutional reforms 
of programs and services. For Study 1, we examined the degree to which survey 
items reflected changes in student perceptions, as projected by institutional 
personnel. For Study 2, we noted relatively large differences in mean student 
responses between successive administrations of a survey, and asked institutional 
personnel to list institutional reforms that might have produced those changes. 
Thus, item sensitivity was examined from two converging perspectives, the first 
based on predicted changes, and the second based on observed differences m the 
data . 

Student Opinion Survey (A -year) ♦ The SOS 4-year was the only instrument 
examined in both Study 1 and Study 2. Both studies provided substancial support 
concerning the sensitivity of this survey to changes in student perceptions 
arising from institutional reform. Hit rates for both studies v;ere over 75%, and 
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the percent of statistically significant hits was over 50% for Study 1. Both 
Section II and Section III of the survey appear to be sensitive to changes in 
student perceptions across a broad range of academic and nonacademic aspects of 
college life. 

Suxrvey of Acadajnic Advising. The use of this survey instrument for 
assessing changes in student perceptions was also supported by the results from 
Study 1- Over 75% of the predicted changes were supported by the student 
response data. Thus, SAA items appear to reflect changes in student perceptions 
resulting from changes in the advising program. However, the results of this 
analysis are based only on the responses of two institutions. Thus, these 
results may not generalize to all SAA user-institutions. 

Withdrawing /Nonre turning Studont Survey. This survey form was examined in 
Study 2 and received relatively strong support. Seventy-eight percent of the 
explanations offered by the five participating institutions were rated 
acceptable. Thus, the analysis supported the validity of the survey for 
reflecting changes in the perceptions of withdrawing students for these 
institutions . 

Adult Loarnar Needs Aflfloggmopt Survey. The analysis of this survey yielded 
somewhat inconclusive results, due to several factors. First, only one 
institution participated in this analysis, thus generalization to other 
institutions is not appropriate. Second, responses to Section III of the ALNAS 
are problematic. For example, one might predict that a particular institutional 
change will lead to a more positive response on the survey form. However, for 
many items, a "more positive response" may be that the students indicate a 
greater need for a particular program, service, or skill (because students become 
more aware, for example, of the complexity of reading comprehension) . For other 
items, a more positive response may be a d ecrease in perceived need because a 
particular program or service has resolved many of the students' needs in those 
areas (for example, learning how to find job openings) . Third, the one 
participating institution administered this survey form to some adult learners 
who were only potential (i.e., not yet enrolled) students, and thus may not have 
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had direct experience with the programs and services assessed by particular 
survey items. Data from these students are therefore suspect. 

Student Opinion Survey (2-year) . This instrument v;as examined in Study 2; 
the data consisted of responses to 10 items from only two institutions. For 
these two institutions, personnel generated acceptable explanations for 70% of 
the differences in student satisfaction, thus supporting the validity of the 
survey for reflecting these differences over time. However, the data are limited 
and this conclusion may not generalize to other institutions. 

Entering Student Survey > The analysis for this survey was based on the 
responses from only three institutions and six items. Too few student responses 
v;ere obtained to permit an accurate interpretation of these data. 
Conclusions 

Generally, the survey items examined in this study showed substantial 
sensitivity to changes in students' perceptions over time. This study suggests 
that several ESS surveys (particularly the SOS 4-year, the SAA, the SOS 2-year, 
and the W/NRSS) can help institutions study the impact of programs and services 
on the perceptions of their student population. 

Factors influencing interpretation. Interpretation of the results of this 
study should be guided by the following considerations and cautions: 

1. Capabilities of institutional personnel. Personnel who provided 
predictions (Study 1) or explanations (Study 2) v;ere not equally 
specific in their responses and differed in both the number and type 
of survey items they believed would be affected by a given 
institutional change. This difference was most noticeable in Study 
1, in which personnel were required to hypothesize relationships 
between institutional change and survey items. 

2. Statistical versus meaningful signif icance • Some changes in student 
perceptions may not have been statistically significant due to small 
sample sizes, but nonetheless may have represented meaningful 
changes. Conversely, some minor differences in student perceptions 
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may have reached st ^tist ical significance and yet may not have 
reflected any meaningful change in student perceptions. Change in 
mean student response must be interpreted in the context of the 
perceptual shift it represents, and whether or not that shift is of 
sufficient importance to warrant further examination . The 
meaningf ulness of a difference in mean student response, regardless 
of the size of that difference, must be determined by institutional 
personnel, not by the statistics associated with it. 
Representativeixess and g«neralizability of roeulta. Three ma^or 
factors affect the generalizability and representativeness of these 
results: (1) the number of participating institutions per survey, 
( 2 ) the number of student responses per survey item, and i 3 } the 
proportion of survey items used as indicators, compared to the total 
number of survey items in the survey. Analyses for each survey 
varied with regard to each of these factors. Generally, the results 
for surveys based on relatively large numbers of institutions, 
student responses , and selected items are more likely to be 
representative of all users than are those based on small numbers. 
Thus, the results of the SOS (4-year) analysis are probably the F.ost 
representative and also most likely to be general izabie to other 
user-institutions, followed by the results for the SAA, the SOS (2- 
year), and the W/NRSS. 
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N.imc 

InNtitiition Nnmc 

Address 

City, State Zip 

DiMr : 

I »im writing to yon concerning ytjur institution's usdgc o\ tlic ACT Slmlent C^pinion Survev, I 
noted from our records tlint your institution h«is used this survey oni» or more timc^ (or the pjsl se\'er.il 
years. I hope tlidt you have found tlie survey data to be Iielpful in identifying key issues for your 
institution. 

An important feature c^f ACTs Evahi.'t ion /Survey Service (MSS) inslnmients is their abihly to 
provide pertinent infomiation about students' perceptions of college. We currentlv pro\'ide Hmiti»d 
ri'liability infoimation about the surveys in our FSS User's (aiide, and hav(» dc»VL'loped normative data 
for several surveys- Because these data are limited, our present research focus is to develop validity d.Ua 
Concerning the informatic>n elicited by tlie surveys. This research will provide data regarding the degree 
to which tlie surveys reflect institutional reform. For example, institutional officials might expect, gi\'en 
certain reforms, that students' responses to related items would change as a result of these reforms. It 
is in regard to this issue that I am writing to you. 

You recently requested copies of the Student Opinion Sur\'ey to be administered to \'(uir students. 
We would like to know if you have implemented reforms or made changes in vour programs or servu'es 
sinc<' your last sur\ev administration that you expert will result in changes in your studt^nts' rL'spv)nses 
on the next administration. Would you please take the time to tell us about these reforms? In addition, 
please tell us the date you initiated the* reforms, the survey items you exp(»ct will be affected, <uid .) brief 
descriptitin of how you expect the respons(*s to change. I have enclosed a response form for your use, 
along with detailed instructions fi^r completing the form. 

I know that time is at a premium for all our uscts, but 1 do hopi' you will bt» «ible to complete tlu» 
fomi I have enclosed. Tlie results of this study will benefit your inslitut'on and ACT by helping us ensure 
iKTurate measures of student perceptions. 

If you have any questicnis or concerns, please* t all Julie Nobie at 31^)/ 337- 1442, collect. On behall 
ot ACT and the colleges that use the F£valuation/Survey Services, thank you in advance tor your generous 
help. 

Sincerely, 



Michael J. Valiga 
Coordinator, Survey Services 
Research Division 
ACT 
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E\MlU(Uit)n /Survey ScTvito 
Survey Repeat ESS Users 

DirecHonb : The purpose of thib survey i:^ to examine the sciiMtivity of ESS surveys to institutic)n<il change. 
Between 1^7^ and 1^86 your in^tihItion participated in ACTS Evalu«Uion/Survey Service (FSS), 
adminibteiing the Survey of Academic Advibing at lea^t once during that lime period. You recently 
requested copies of the survey to be ndmjniston»d on your campus. 

IMease begin bv writing your name and phone number in the spaces provided. (We would Hke to be abU* 
to cal! you if we need further clarification.) TIk'u, identify any changes or reforms, and their dale of 
initiation, that you think might influence your students' responses on the next administration, Tlu'se 
changes could include, for example, one or more of the folkm'ing: curriculum refonns, changes in 
marketing strategies, policy adjustments, changes in financial status, or changes in the stnident ptipulation, 
rieas(» be as specific as possible in describing the reforms. After describing the reforms, please identify 
the specific survey item(s) that you expect will be influenced by the reform(s), and the direction in which 
you expect students' responses to change. 

An example has been provided to supplement these directions. If you have additional questicms, please 
call julie Noble (collect) at 319/337-1442. rieas(» return this questionnaire in the enckised envekipe by 
April ]5 before you receive the results of the next survey administration. Tiv^iUk you very* much for your 
cooperation. 
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livaluatioa'Survey Service 
arvey of Repeat ESS Users 



( Name: 
( dlloj^c Name: rhiuio: 



■inr\'('y: 



Description of reform/change 


Section/item # you feel 


Brief description of 


niiLi llliil<llcLl 


will L/C (t 1 1 cLlcTLI 




I x.miple: Student Opinion Siirvc^v 


Section ill item 30 


Student resp{)i^st's are e\pet ted 




item :n 


tt) he muel^ more po->ili\H' lh<\n in 




item 33 


the past. 


ImH, IS\S6 Implement eel cempTiterized 






rej;istnitic)n procedures. Lines «ire much 






. shorter; the registration process takes 






! much less time. 






; Description of reform/chanjj,e 


Section/item # you feel 


IJrief description of 


I and date initiated 


will be affected 


expected change 


! 
1 

1 
1 

i 

i 
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Table Bl 



Demographic Information for Institutions Using the Student Opinion Survey (4-year) - Analysis of 
Projected Differences 



Institution 


Region 
of U.S. 


Affiliation 


Size of 
Community 
(1,000's) 


Number of 
students 
(1,000's) 


Principal majors 
and percent 
of students 


A 


South 


Public 


10-50 
50-1 (X) 


1-5 


Education 
Business 


35';? 


B 


Central 


Private 


10-50 


0-1 


Health 


100'/? 


C 


Central 


Private 


50-1 (X) 


1-5 


Business 
Soc. Sci. 


2{)'Z 


D 


N. Cent. 


Private 


10-50 


1-5 


Business 


30';? 


E 


Central 


Private 


10-50 


1-5 


Business 
Education 


liVZ 


F 


East 


Public 


10-50 


5-15 


Business 
Math 


257, 
]47 
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Table B2 



Sampling Procedures for Institutions Using the Student Opinion Survey (4-year) - Analysis of Projected 
Differences 



Institution 


Sampling component 


Administration 1 


Administration 2 


A 


Administration dates 
Number of surveys returned 
Sample type 
Administration mt)de 
Sample ccimposition 
Response rate 


04 /S5 
754 

Random 
In class 

All four undergraduate classes 
95% 


04/87 
642 

Random 
In class 

All four undergraduate classes 
97% 


B 


Administration dates 
Number of surveys returned 
Sample type 
Administration mode 
Sample composition 
Response rate 


()4/S6 
112 

Whole population 
In class 

Juniors and Seniors 
86% 


05/87 
112 

Whole population 
In class 

Juniors and Seniors 
100% 


C 


Administration dates 
Number of sur\'eys returned 
Sample typo 
Administration mode 
Sample composition 
Response rate 


03/86 
201 

Random 

Several methods 

All four undergraduate classes 

40% 


03/87 
182 

Random 

Several methods 

All four undergraduate classes 

36% 


D 


Administration dates 
Number of surveys returned 
Sample type 
Administration mode 
Sample composition 
Response rate 


01 /H4 
138 

Random 

Several methods 

All four undergraduate classes 

/ Xh 


01/87 
125 

Random 

Several methods 

All four undergraduate classes 

fc 


E 


Administration dates 
Number of surveys returned 
Sample type 
Administration mode 
Sample composition 
Response rate 


04/86 
270 

Whole population 
In class 

All four undergraduate claLses 
100% 


04/87 
242 

Whole population 
In class 

All four undergraduate classes 
100% 


F 


Administration dates 
Number of surveys returned 
San^ple type 
Administration mode 
Sample composition 
Response rate 


04/85 
677 

Random 
U.S. mail 

All four undergraduate classes 

647r 


03/87 
685 

Random 

Several methods 

All four undergraduate classes 

55% 
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Table B3 



Student Characteristics for Institutions Using the Student Opinion Survey (4-year) - Analysis of 
Projected Differences 



Institution 


Student characteristic 


Administration 
1 


Administration 
2 


A 


Under age 30 




897 




Race: Caucasian 
Black 


21% 


O 1 h 




Percent males 


44% 


40% 




Percent unmarried 




83% 




Purpose: A. A. degree 
B.A. degree 
M.A./Ph.D 


06 7r 

73% 

04 7r 


0/ Yf 

71 7 
06% 




Residence: Dorm 

Off-campus 


OL /c 
437r 


"t\ \ /( 

557 


B 


Under age 30 


817 


oJ 7 




Race: Caucasian 
Black 


947 

037' 


047 




Percent males 


057 


027 




Percent unmarried 


7(S7 


707 




Purpose: B.A. degree 


1007 


1007 




Residence: Dorm 

Off-campus 


227 
11% 


197 

/C\/( 


C 


Under age 30 


100% 


92% 




Race: Caucasian 
Black 


967 
017. 


957 
02% 




Percent males 


35% 


39% 




Percent unmarried 


99% 


917 




Purpose: B.A. degree 
M.A./Ph.D 


967 
02% 


85% 
12% 


1 

i 
i 

i' 


Residence: Dorm 

Off-campus 
Frat/ Sorority 


797 
037 
187 


61% 

32% 
077 



continued on next page 
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Table B3 (continued) 







Administration 


Administration 


Institution 


aruucnt cnrtraciciisiic 


1 

1 




D 


Under age 30 








Race: Caucasian 




95% 




Black 




02% 




Percent males 


DZ> n- 






Percent unmarried 








Purpose: B.A. degree 


79^;? 


75% 




Ivi . A. / 1 n.u 


in /r 


O^ /(• 




Transfer credits 


06^ 


05% 




Residence: Dorm 


56% 


54%. 




Off-campus 


k 


'Tu /r 


E 


Under age 30 








Race: Caucasian 




83% 




Black 




12% 




Percent males 


36% 


43% 




Percent unmarried 


40% 


44% 




Purpc^se: B.A. degree 


/c 


y\i ft 




M.A./Ph.D. 


01% 


00% 




Certification 




04% 




Residence: Off-campus 




96% 


F 


Under age 30 








Race: Caucasian 


93% 


91% 




Black 


01 % 


02% 




Percent males 


36%: 


31% 




Percent unmarried 


95^;? 


93% 




Purpose: B.A. degree 




92% 




M.A./Ph.D 


01% 


02^;? 




Residence: Dorm 


61% 


42^;j 




Off-campus 


37% 


55% 
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Academic Advising - Study 1 



27 



Table CI 



Sampling Proc^^dures for Institutions Using the Survey of Academic Advising - Analysis 
of Projected Differences 



Institution 


Sampling component 


Administration 1 


Administration 2 


A 


Administration dates 
Number of surveys returned 
Sample type 
Administration mode 
Sample composition 
Response rpte 


04/86 
176 

Whole population 
U.S. mail 
S<iphom()res only 
U% 


( )4 / o/ 
147 

Whole population 
U.S. mail 
Sophomores only 
37% 


B 


Administration dates 
Number of surveys returned 
Sample type 
Administration mode 
Sample composition 
Response rate 


04 /86 
277 

Whole population 
Other 

Freshmen only 

747r. 


02/87 
281 

Whole population 
Other 

Freshmen only 

707r 
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Table C2 

Student Characteristics for Institutions Using the Survey of Academic Advising - Analysis of Projected Differences 



Institution 


Student characteristics 


Administration 1 


Administration 2 


A 


Under age 30 


737 


7)S7( 




i erCUnt V_c1utdblclil 


947, 


977 




Percent males 




317 




LjI a: j.u to 4,u 


AV7r 


477' 




2.0 to 2.9^ 




507 




oeuHV z.u 


m '7, 


037 




Purpose: A. A. degree 


69'/ 


627 




Transf er credits 


19'/ 


237 




Self improvement 




057 




Certification 


03'/ 


027. 


B 


Under age 30 


1007r 


1007. 




Percent Caucasian 


90'/ 


937 




Percent males 


327 


387 




CPA: 3.0 to 4.0 


287 


317 




2.0 to 2.99 


537 


497 




below 2.0 


197 


19% 




Purpose: B.A. degree 


897 


87% 




Uncertain 


047 


04% 
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Table Dl 

Student Characteristics for One Institution Using the Adult Learner Needs Assessment Survey - Analysis 
of Projected Differences 





Administration 


Student characteristics 


1 


2 


3 


Age 23-30 




267r 


287. 


Percent Caucasian 


887r 


877r 


917 


Percent males 


^27, 


imc 


347 


Marital jjtatus: Single 




257. 


227 


Married 






477 


Divorced 


12% 


087 


207 


Highest level 








of education: High school 




137 


197 


Prebaccalaurea te 




457 


497 


Postbaccalaureate 


54^ 


39% 


217 



Table D2 

Sampling Procedures for One Institution Using the Adult Learner Needs Assessment Survey - Analysis 
of Projected Differences 





Administration 


Sampling component 


1 


2 


3 


Administration dntc^ 


06/85 


03 /Hh 


03/87 


No. of surveys rcturnt'd 


108 


84 


106 


Sample* type* 


Random sample 


Random sample 


Random sample 


Administration m(^dc 


U.S. mail 


U.S. mail 


Several methods 


Sample composition 


Enrolled adiiUs 


Potential /aclual 
adult students 


Graduates <fe 
undergraduates 


Response rate 


43';? 


40*;^ 


A^7< 
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Name 

Institution Ndmc 
Address 
Citv, State 



I 
I 

7,p I 



DcMr 

I im\ writing to yon concerning your in.stitution's usage of the ACT Hntering Student Survey. I noted from J- 
records that your institution has used this survey one or more times for the past several years. I h(ipe that you h.ve 
found the survey data to be helpful in identifying key issues for your institution. w 

An important feature of ACT's Evaluation/Survey Service (ESS) instruments is their ability to provide pertiiBt 
information about students' perceptions of college. We currently provide limited reliability information about the sur\^ 
in our ESS User's Guide, and have developed nomiative data for several surveys. Because these data are limited, lur 
present research focus is to develop validity data concerning the information elicited by the surveys. This research B 
provide data regarding the degree to which the surveys reflect institutional reform. For example, institutional officB 
might expect, given certain reforms, that stxidents' responses to related items would change as a result of these refoms. 
It is in regard to this issue that I am writing to you. M 

We have, in our ESS files, data from several institiititins that have administered the same ESS surveys more tH 
once over the last few years. In examining these data, we have noted relatively large differences over time in studeits' 
responses to specific items. These differences are not consistent across institutions or across items. In addition, t[ 
cannot be attributed to differing types of samples, administration techniques, (ir sampling methods, as we selected 
those colleges with similar samples over time. 

We have decided, therefore, to survey ESS participants to learn more about why these differences niightbe 
occurring. I have enclosed a response form listing the survey items that, based upon your survt^v data, have shcMj 
relatively large changes in mean student response over time. Would you please take the time tci tell us why you tli^ 
these differences might be occurring? Detailed instructions for completing the form are enclosed. 

I know that time is at a premium for all our users, but 1 do hope ytui will be able to complete the form I h 
enclosed. The results of this study will benefit your institution and ACT by helping us ensure accurate measure] 
student perception. 

If you have any questions or concerns, please call Julie Noble at 3N/337-1442, collect. On behalf of ACT Av} 
the colleges that use the Evaluation /Survey Services, thank you in advance for your generous help, 

Sincerely, 



I 



Michael J. Valiga 
Coordinator, Survey Services 
Research Division 
ACT 



I 

I 



I 
I 

3D ■ 



Evaluation/Survey Service 
Validity Study Survey 



Directions! The purpose of this survey is to examine the sensitivity of ESS burveys to institutional change. Between 1979 
and 1986 your institution participated in ACTs Evaluation/Survey Service (ESS), administering the Entering Student 
Survey at least twice during that time period. The survey and the dates of administration are identified on the attached 
response form. Administration dates were limited to those occurring in a minimum of one year inaements; data were 
used from up to five survey administrations. A maximum of five items are listed that have been identified as having 
relatively large mean response differences over time. Tl-^e response means are reported under each date of administration. 
The response means for the Entering Student Survey are computed such that 4 = very important and 1 = not impcirtant, 
and 5 = strongly agree and 1 = strongly disagree. In some cases, means will be reported as a blank or a This will 
occur if the sample sizes were insufficient to provide reliable data for this study. 

Please begin by writing your name and phone number in the spaces provided. (We would like to be able to call you if 
we need further clarification.) Then, for each iteu listed, please identify any institutional changes that might have 
contributed to the differences in mean student responses over time. These changes could include, for example, one or 
more of the fo!lowing: curricular reforms, changes in marketing strategies, policy adjustments, changes in financial status, 
or changes in the student population. Please supply as much information as possible regarding these changes. 

An example has been provided to supplement these directions. If you have additional questions, please call Julie Noble 
(collect) at 319/337-1442. Please return this questionnaire in the enclosed envelope by August 1. Thaiik you very much 
for your cooperation. 
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EVALUATIOWSURVEY SERVICE 
VALIDITY STUDY SURVEY 



CODE: 

COLLEGE NAME: 

SURVEY: ENTERING STUDENT SURVEY 



YOUR NAME: 
PHONE: 



1 



EXAMPLE: 

SECTION III ITEM 30; 
STUDENT OPINION SURVEY 

A. GENERAL REGISTRATION PROCEDURES 



RESPONSE MEAN 



02/83 


03/84 


03/85 


03/86 


2.67 


2.75 


3.25 


3.40 



1 



Fall, 1984-implemented computerized registration procedures resulting in much shorter registration lines aiu 
faster registration procedures. 



RESPONSE MEAN 



A. ENTRANCE REQUIREMENTS FOR THE 
COLLEGE 



09/82 


12/83 


09/84 






2.13 


2.32 


2.46 







Briefly summarize any reforms /changes you have implemented or experienced at your inshtution that you M 
could have contributed to these changes in mean student responses over time. Please be as specific as possi ' 
in outlining this information; any additional evidence would be appreciated. Also include the dates th 
changes /reforms took place. 



I 

I 
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EVALUATION/SURVEY SERVICE 
VALIDITY STUDY SURVEY 



PARKING FACILITIES AND RESPONSE MEAN 

SERVICES 



04/82 


04/83 


01/85 


03/86 




3.19 


3.52 


3.71 


3.91 





Briefly suminarizG any reforms/changes you have implemented or experienced at your institution that you feel 
could have contributed to these changes in mean student responses over time. Please be as specific as possible 
in outlining this information; any additional evidence would be appreciated. Also include the dates these 
changes/reforms took place. 



LABORATORY FACILITIES RESPONSE MEAN 



04/82 


04/83 


01/85 


03/86 




3.56 


3.86 


3.99 


4.01 





Briefly summarize any reforms /changes you have implemented or experienced at your institution that you feel 
could have contributed to these changes in mean student responses over time. Please be as specific as possible 
in outlining this information; any additional evidence would be appreciated. Also include the dates these 
changes/reforms took place. 
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EVALUATION/SURVEY SERVICE 
VALroiTY STUDY SURVEY 

THIS COLLEGE IN GENERAL RESPONSE MEAN 



04/82 


04/83 


01/85 


03/86 




4.18 


4.13 


4.28 


4.43 





Briefly summarize any reforms/changes you have implemented or experienced at your institurion that yo 
could have contributed to these changes in mean student responses over time. Please be as specific as po 
in outlining this information; any additional evidence would be appreciated. Also include the dates 
changes/reforms took place. 



OVERALL IMPRESSION OF THE QUALITY 

OF EDUCATION AT THIS SCHOOL RESPONSE MEAN 



04/82 


04/83 


01/85 


03/86 




4.07 


4.06 


4.19 


4.37 





Briefly summarize any refomis/ changes you have implemented or experienced at your institution that y 
could have contributed to these changes in mean student responses over time. Please be as specific as p 
in outlining this information; any additional evidence would be appreciated. Also include the date 
changes /reforms took place. 
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APPENDIX F 



Demographic Information and Sampling 
Procedures Used for Participants in 
Study 2 



44 

o 

ERIC 



Table Fl 



Demographic Information for Institutions Using ESS Surveys - Analysis of Explained Differences 





SOS (4-year) 


W/NRSS 


SOS (2-year) 


ENSS 


Demographic characteristic 


(K = 21)' 


(K = 5) 


(K = 2) 


(K = 3) 


Roj^ion of US: 










Pacific Mountain 


2 


1 


0 


0 


North Central 


5 


I 


0 


0 


Great Lakes 


4 


1 


0 


1 


South Central 


4 


0 


1 


0 


South Atlantic 


2 


1 


0 


2 


Mid Atlantic 


4 


1 


1 


0 


New England 


0 


0 


0 


0 


Affiliation: 










Public 


8 


4 


2 


0 


Private 


3 


0 


0 


0 


Religious 


10 


1 


0 


3 


Highest degree offerc^d: 










Ph.D. 


5 


0 


0 


0 


M.A. 


9 


3 


0 


1 


B.A. 


6 


1 


0 


2 


A.A. 


1 


1 


2 


n 


Size of community (in KKKr^i): 










Under 10^ 


5 


1 


1 


1 


l()-5() 


8 


4 


1 


0 


50-1 (K) 


4 


0 


0 


1 


l(K)-5()() 


0 


0 


0 


0 


Over 5(K) 


4 


0 


0 


1 


Total enrollment (in KKKI's): 










Less than 1 


7 


1 


1 


2 


1-5 


8 


3 


(1 


1 


5-15 


4 


1 


(1 


0 


Over 15 


2 


0 


1 


0 


Principal majors offered: 










Social Srif^ncf's 


2 


0 




1 


Business 


1o 


2 




2 


Math-Science 


1 


1 




0 


Health 


5 


1 




(1 


Education 


5 


1 




1 


Percent nf students in major (median): 










Social Sciences 


25 


0 




25 


Business 


28 


24 




2S 


Math- Science 


28 


37 




0 


Health 


46 


55 




0 


Education 


26 


33 
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K = total number of instituticms that responded for each survey 
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